Leveraging emotional transitions in media to modulate emotional impact of secondary content

ABSTRACT

Systems and methods for determining emotional responses to videos, generating emotional arc based on the responses and reactions, and determining the type and/or point of insertion of secondary content based on the determined emotional responses are disclosed. The methods display primary content to a plurality of users and determine their emotional response as they consume the primary content. An average emotional arc is generated based on the determined emotional response and it is compared to a current emotional arc generated based on emotional response from a current user. When values of the current emotional are within a predefined range of the average emotional arc at a particular play position in the timeline of the primary content, a secondary content that is designated for the value of the current emotional arc is inserted in the primary content at the particular play position.

FIELD OF INVENTION

Embodiments of the present disclosure relate to generating emotionalgraphs and arcs based on emotional responses to consumption of primarycontent and determining which type of secondary content and its point ofinsertion in the primary content produces a desired emotional impact.

BACKGROUND

In the current climate where technological advances place access toenormous amount of content at users' fingertips, individuals' attentionspans are getting shorter and shorter. People have several windows,tabs, and applications open on their mobiles and laptops and jump fromone application to another within seconds. When streaming media, such ason streaming media platforms like Netflix™, Hulu™, HBO™ or Disney™, datahas shown that individuals jump from one episode to another constantlylooking for content that grabs their attention. Even when a series ofepisodes exists, such as “Game of Thrones,” individuals skip fromepisode to episode and even fast-forward through a lot of content withinan episode looking for something that interests them.

In such a world, where attention-grabbing is challenging, contentproviders, publishers, advertisers, etc. are constantly faced with adilemma of how to grab a user's attention for particular content, whichin the case of advertisements, may be inserted during playback of other(primary) content. Often several attempts are made at capturing theuser's attention such that the advertisement generates the desiredimpact. These efforts may include referral marketing, clickthroughmarketing, and marketing based on content previously selected orsearched by the user. For example, if a user has searched for campingequipment, such search data may be captured and used by publishers oradvertisers to provide camping-related advertisements during contentbeing consumed by the user. As such, advertisements related to campingmay show up in the middle of a movie or show being consumed or during atimeout of a live-streamed game. Likewise, if a user has searched forwedding gowns, advertisers may provide supplemental content related towedding gowns or other wedding-related accessories to be displayed tothe user while they are consuming on-demand or live broadcast mediaassets.

Providing supplemental content based on the user's search history isuseful, but generally does not adequately consider the user's emotionalstate when the content is provided or consumed (e.g., consumed, listenedto, and/or read). For example, if a user is watching a sad scene in aprimary content, perhaps of a very emotional scene where somebody isdying, or some other moment that sets a sad or somber mood, displayingsupplemental content for a wedding gown would likely not be appealing(or as appealing) to that user at that time.

Current content-insertion methodologies often do not consider howinserted or supplemental content emotionally relates to the primarycontent being consumed. Current methodologies also do not track theuser's emotional response and changes in the user's emotional state asthey consume different scenes or segments of the media asset. As such,secondary or supplemental content that has nothing to do with theprimary content being consumed and/or nothing to do with the user'semotional state are displayed to the user at random times during theconsumption of the media asset. Such consumption of content may have lowimpact and commonly results in the user focusing away from the contentor switching to a different channel, for instance.

As such, there is a need for a method and system to take into accountthe content being consumed, the type of secondary content, and thestages of the user's emotional responses such that secondary contentthat fits with the user's emotional state is displayed at the right timeto create a higher level of impact.

BRIEF DESCRIPTION OF THE DRAWINGS

The various objects and advantages of the disclosure will be apparentupon consideration of the following detailed description, taken inconjunction with the accompanying drawings, in which like referencecharacters refer to like parts throughout, and in which:

FIG. 1 is a block diagram of an example process for comparing anestimated emotional arc with an actual emotional arc to determine a typeof secondary content to insert in primary content for its highestimpact, in accordance with some embodiments of the disclosure;

FIG. 2 is a block diagram of an example process for generating anaverage emotional arc, in accordance with some embodiments of thedisclosure;

FIG. 3 is a block diagram of an example process for gauging a user'semotional response to a first part of secondary content and adjustingthe second part based on the user's response, in accordance with someembodiments of the disclosure;

FIG. 4 is a table of options for analyzing emotional responses relatingto the primary content and secondary content for determining impact ofsecondary content, in accordance with some embodiments of thedisclosure;

FIG. 5 is a block diagram of various components of a system fordetermining emotional arcs and inserting secondary content, inaccordance with some embodiments of the disclosure;

FIG. 6 is a block diagram of communications between different componentsof the system for determining emotional arcs and inserting secondarycontent, in accordance with some embodiments of the disclosure;

FIG. 7 is a block diagram of an example system for determining emotionalresponses for content, generating associated emotional arc, anddetermining insertions of secondary content, in accordance with someembodiments of the disclosure;

FIG. 8 is a block diagram of a user's media device, in accordance withsome embodiments of the disclosure;

FIG. 9A is an example of an activation graph, in accordance with someembodiments of the disclosure;

FIG. 9B is an example of a valence graph, in accordance with someembodiments of the disclosure;

FIG. 9C is an example of an emotional arc, in accordance with someembodiments of the disclosure;

FIG. 9D is an example of a table that represents values of the emotionalarc, in accordance with some embodiments of the disclosure;

FIG. 10 is an example of a list of emotions and their variances frommild to intense and pleasant to unpleasant, in accordance with someembodiments of the disclosure;

FIG. 11 is an example of a list of emotions and their variances fromactivated to deactivated and pleasant to unpleasant, in accordance withsome embodiments of the disclosure; and

FIG. 12 is graph of positive and negative movements in a valence andactivation graph, in accordance with some embodiments of the disclosure.

DETAILED DESCRIPTION

In accordance with some embodiments disclosed herein, techniques areprovided that help to overcome some of the above-mentioned limitationsby comparing an average emotional arc with a current emotional arc. Theaverage emotional arc is generated based on emotional responses from aplurality of users that have previously consumed primary content, andthe current emotional arc is based on real-time emotional responses by acurrent user consuming the primary content. The disclosed techniquesdetermine, based on the comparison, either the type of secondary contentto insert, a play position in the primary content to insert thesecondary content, or both.

As referred to herein, the terms “content” and “media content” may beunderstood to mean electronically consumable user assets, such astelevision programming, as well as pay-per-view programs, on-demandprograms (as in video-on-demand (VOD) systems), advertisements(including infomercials, commercials, and other consumable assetsassociated with marketing a product or a service), Internet content(e.g., streaming content, downloadable content, Webcasts, etc.), videoclips, audio, content information, pictures, GIFs, rotating images,documents, playlists, websites, articles, books, electronic books,blogs, chat sessions, social media, applications, games, and/or anyother media or multimedia and/or combination of the same. As referred toherein, the term “multimedia” should be understood to mean content thatutilizes at least two different content forms described above, forexample, text, audio, images, video, or interactivity content forms.Content may be recorded, played, transmitted to, processed, displayedand/or accessed by user equipment devices, and/or can be part of a liveperformance. The present disclosure further contemplates thatembodiments discussed below with respect to particular primary and/orsecondary content, e.g., video content, are also applicable to otherforms of primary/secondary content, e.g., audio, images, text,multimedia, and the like.

In one embodiment, primary content, e.g., primary video, is displayed toa plurality of users and their emotional response data on a per scene,frame, or segment over the duration of the primary content is obtained.The emotional response data is then used to generate an emotional arcfor each user, which is then used to generate an average emotional arcfor the plurality of users. The emotional response data are collectedthrough a variety of components, such as cameras, heartbeat detectioncomponents, sensors, lidar, gyroscopes, and accelerometers. Thesecomponents monitor the user's gaze and other biometrics data as the userconsumes the primary content asset and determine the user's emotionalresponses. For example, cameras located in devices may track the user'seyeball movement to determine if the user's gaze is directed towards theprimary content. The cameras may also be used to track specificallywhich portions, frames, and segments of the primary content were gazed.Likewise, sensors in smart watches, mobile phones, and other wearablesmay determine the user's heartbeat to determine the user's emotionalstate, e.g., fast heartbeat may be indicative of excitement and so on.These emotional states experienced during the consumption of the primarycontent are represented using Russell's circumplex model of emotions astwo numbers: one number for activation and one number for valence.

The average emotional arc for the primary content is used to generate anestimate for a current user, i.e., a prediction that the current userwatching the same primary content is likely to have same or similaremotional reactions to the various scenes and segments of the primarycontent as did the other users who previously consumed the primarycontent. In other words, the average emotional arc is assigned as theestimated emotional arc for a particular user and adjustments andoffsets to the estimate are made based on actual emotions that aresubsequently determined. The adjustments and offsets, as describedbelow, are made to the valence and activation values. For example, theestimated emotional arc is comprised of activation and valence values.If an actual activation and valence value at a certain play position inthe primary content differs from the estimated activation and valencevalue for that play position, then the activation and valence values areadjusted by applying an offset that is equivalent to the variationdetermined.

The average emotional arc is used as an estimating tool and compared tothe actual emotional response of the user that has consumed the primarycontent. The actual emotional response may be in the form of a currentuser's emotional arc. The comparison is performed to determine if theactual response from the user is as estimated. If it is not, then theaverage emotional arc is adjusted by applying an offset based on theamount of variance of the actual emotional response from the averageemotional arc.

In some embodiments, supplemental content, e.g., secondary content, isintended to be inserted at a particular average emotional arc value.Since the average emotional arc is generated based on activation andvalence values, the secondary content may be designed to be insertedwhen the activation and valence values in the average emotional arcreach the desired value. If the system determines that the values in theaverage emotional arc have reached the desired value to insert thesecondary content, a secondary check may be performed. This secondarycheck includes comparing the user's emotional arc with the averageemotional arc to ensure that the actual emotional values are within arange of the estimated emotional value, i.e., within the range of thedesired value for inserting the secondary content. If the actual valuesare within the estimated and desired range then secondary contentdesignated for the range is inserted into the primary content anddisplayed to the current user.

In one embodiment, where the user's emotional arc is not within therange of the emotional values of the average emotional arc, then thevalues of the user's current emotional arc at the current play positionin the primary content are used to find secondary content, or a versionof the secondary content, that is designated for the emotional valuesthat are within a range of the user's current emotional arc values. Tothe extent such secondary content is not available, then secondarycontent that is closest to the value of the user's current emotional arcat the current play position is inserted into the primary content.

Several combinations of determining emotional responses, generatingemotional arc corresponding to the emotional responses, comparing theemotional arcs, and monitoring in real-time the consumption by thecurrent user are disclosed in the embodiments described herein. Theseinclude using a first plurality of users to get an emotional responsefor the primary content and a different second plurality of users to getan emotional response for the secondary content. The embodiments alsoinclude monitoring the emotional arc of the primary content andinserting secondary content at any play position in the primary contentwhen the desired value for the emotional arc in the primary content isachieved. In other words, achieving the desired value for the emotionalarc is associated with the user's emotions reaching a state that isdesired or optimal for inserting the secondary content. For example, adesired value may be associated with the feeling of excitement. Ifparticular secondary content is better received and reinforced when auser is excited, then the content provider would look for the desiredvalue for the emotional arc in the primary content to reach a valueassociated with excitement such that the secondary content can beinserted at that point in time.

In one embodiment, flexibility as to where secondary content can beinserted is provided while in other embodiments the insertion of thesecondary content is restricted to specific play position or acommercial break in the primary content whenever such breaks are taken,such as in a live game etc.

In another embodiment, a determination is made as to which type ofsecondary content is to be inserted based on the emotional responsevalues. These and additional embodiments, processes, and systems andcomponents to determine which secondary content to insert and at whatplay position are described below.

FIG. 1 is a block diagram of an example process for comparing an averageemotional arc with an actual emotional arc of the user to determine atype of secondary content to insert in primary content for its highestimpact, in accordance with some embodiments of the disclosure.

In some embodiments, as depicted blocks 103-104B of FIG. 1 , the pointsof insertion for secondary content may be predetermined at specific playpositions in the primary content. For example, in a movie or a talk showbroadcast, commercial or advertisement breaks may be predetermined atcertain intervals, such as after every five or 10 minutes in theprogram. In another example, such as in a news broadcast, or a footballor a basketball game, the points of insertion for secondary content maybe based on an occurrence of an event, such as whenever the newscastends a particular segment, such as the weather segment, or in the sportssetting whenever a team calls for a time out. The secondary content mayneed to be inserted in these predetermined positions, and there may belittle or no flexibility in where to insert them. In such scenarios, aswill be further described in relation to FIG. 2 below, the system maydetermine the most optimal type of secondary content to insert in thepredetermined position such that the type of secondary content selectedproduces the highest possible impact for the inserted secondary content.

In other embodiments, the points of insertion for secondary content intoprimary content may be flexible. In such embodiments, the emotional arcof the primary content may be monitored and when the value of theemotional arc reaches a desired value, the secondary content may beinserted.

In some embodiments, the different types or secondary content that canbe inserted may include similar messaging but may be presented indifferent tones and emotions to enhance their impact. For example, amore energetic tone may be used for the secondary content for the sameproduct than a subtle tone if the control circuitry determines that theenergetic version of the secondary content may resonate more with aviewer and evoke a desired emotion that will result in having thehighest impact for the secondary content. The control circuitry may makesuch determinations based on analyzing various factors. In otherembodiments, the different types of secondary content may includedifferent messaging and may be presented in different tones and emotionsto enhance their impact.

In one embodiment, the process of FIG. 1 may begin by determining anaverage emotional arc of the primary content at block 101. Averageemotional arc, which is also referred to herein as estimated emotionalarc or default emotional arc, is a tool that can be used to estimate thecurrent user's likely emotional arc, i.e., how likely it is that thecurrent user will act in the same or similar manner as the averagenumber of users did in response to watching the primary content. In oneembodiment, the primary content may be an on-demand media asset, and inanother embodiment, the primary content may be a live broadcast.

In some embodiments the emotional arc of the content may be determinedas described in FIG. 2 . As depicted at block 201, emotional data from afirst plurality of users is obtained. The emotional data relates to eachuser's emotional response to a display of primary content. The emotionaldata is then used to generate an emotional arc that aggregates thechanging emotional states of the viewers/users of the primary contentfrom scene to scene or segment to segment from the start to the end ofthe primary content.

The process of gathering the emotional responses includes sending avideo stream that includes a primary content asset to the firstplurality users. As the first plurality of users watch the display ofthe primary content on a display device, emotional response data, e.g.,biometric data from the display device is collected. Such biometric datais indicative of the users' emotions in response to consuming thedisplayed primary content. The biometric data, including any gaze data,can be determining continuously thorough the playing of the primarycontent, or can be determined at fixed intervals, such as at every 15seconds, or at the end of each segment. Although biometric data isdescribed above as data used to derive emotional response, theembodiments are not so limited and they may include other data, such asbrain activity, facial expressions, and survey data, to derive emotionalresponse.

The emotions and the various emotional states experienced during theconsuming of the primary content may be represented using Russell'scircumplex model of emotions as two numbers: one number for activationand one number for valence, for example, as depicted in FIGS. 9A-9D. Theactivation (or intensity) value indicates how energized the user iswhile watching the displayed content, such as a segment of the primarycontent. As referred to herein viewer and user are used interchangeablyand mean the same. The valence value indicates how positive or negativethe feelings are, such as the degree of positive or negative emotion orfeeling. Each dimension or value of activation and valence can be givena fixed scale, e.g., −1 to +1 or some other predetermined scale.Alternatively, each dimension or value of activation and valence can begiven a scale of 0-30, as depicted in FIG. 9D, or any other desiredvalue.

The activation and valence value may change throughout the consumptionof the primary content and even within each segment of the content. Forexample, an exciting car chase may lead to high activation and eitherpositive or negative valence depending on whether the protagonist isabout to make a deadline or is being chased by a serial killer.Similarly, at the end of a thriller, when the protagonists areexplaining how they won, the activation could go down whereas thevalence could become quite positive.

The biometric data and gaze data may be collected frame by frame or asegment by segment on a periodic basis, or at certain key points in theprimary media asset. Other alternatives to biometric data, such as brainactivity, facial expressions, and survey data, may also be used toderive emotional data. The biometric data, such as heart rate, gaze andgalvanic skin response may be collected via the display device's camera,heart rate monitor, sensors, and other components.

In one embodiment, as the primary content transitions from plot to plot,frame to frame, segment to segment, or one time interval to another timeinterval, the biometric data is captured for each such plot, frame,segment or interval and stored, for instance, in a storage of thedisplay device. In another embodiment, such biometric data may be sentto a server for processing.

The biometric data differs from user to user as their emotionalresponses to a scene, frame, or segment of the primary content maydiffer. For example, one user may find a scene in the primary content tobe a sad scene while another user may not find that scene to be sad.Likewise, one user may find a scene to be exciting, funny orexhilarating, while the other may find the scene to be an average sceneor not exciting.

The degree of the biometric data also differs from user to user. Forexample, even when one user may have a similar emotional response toanother user, the degree of the response may vary. For example, inresponse to a sad emotional scene in the primary content, one user mayfeel sad to the degree of crying while the other user may also feel sad,but not as much. Or a user may have seen the same scene multiple times,or many similar scenes, thereby making the scene more repetitive ormundane and not invoking the same emotional response as in another userwho may be consuming the scene for a first time. In this regard, thesystem may determine whether the user has consumed the primary content,or a specific scene in the primary content, for the first time or isconsuming it for the second or a subsequent time. The system may trackand store such data relating to the number of times the same content hasbeen consumed. The system may provide an offset in an emotional responseif the primary content or scene has been consumed more than once. Theoffset may be larger for each time the same content is consumed tooffset the mundane or muted affect it may cause on the user due torepetitive consumption. Whatever the reasons for the different types anddegrees of emotional response, the biometric data specific to each usermay be captured and used for processing.

At block 202, the biometric data from each user, from a first pluralityof users that have previously consumed the primary content, may be usedto generate a graph, also referred to as an emotional arc, for eachuser. In one embodiment, biometric data collected for the four usersdepicted in block 201 may be used to generate a customized emotional arcfor each of the four users in block 202. The emotional arc is depictedover a period of time and each data point in the emotional arc relatesto a play position in the primary content.

The emotions of the user, also referred to as the user's emotionalstate, in one embodiment, is represented using Russell's circumplexmodel of emotions as two numbers; one number for activation (also calledintensity) and one number for valence. The emotional arc representsthese emotional states experienced by the user during the display of theprimary content or a segment of primary content, such as emotionsexperienced while different scenes and emotional transitions in theprimary content are displayed to the user.

At block 203, a control circuitry or a system, such as the systemdepicted in FIG. 7 , is used to calculate an average of all theemotional arcs from block 202 that were generated based on the biometricdata of the first plurality of users. In one embodiment, the average maybe a collective average of emotional arcs of all the users that haveconsumed the primary content. In another embodiment, the first pluralityof users may be further categorized based on, for example, their age,gender, location, and profession, and an average may be taken for eachcategorized group. The first plurality of users may also be categorizedbased on number of times they consumed the primary content. In yetanother embodiment, the first plurality of users may be categorizedbased on the time they consumed the primary content, such as duringmorning, afternoon, night, or during weekdays, or weekends. Separateaverages may be computed based on each group categorized.

In one embodiment, the emotional arc score from block 203 may be relatedto a type of emotion or a combination of emotions. For example, it maybe related to an emotion of sadness, calm, contentment, excitement, orany of the emotional examples as depicted in FIGS. 10 and 11 . It mayalso be a combination of emotions with different intensities andactivation values.

In brief, stated another way, to obtain the average emotional arc asdepicted in block 101, the system may cause the control circuitry to a)displaying the primary content to a plurality of users, b) obtainingtheir emotional response, which is based on their biometric data duringtheir consumption of each frame, segment, scene, or some other portionof the primary content, c) determining an emotional arc for each user,and d) calculating an average emotional arc for all the users who haveconsumed the primary content. This approach may be applied to primarycontent that is an on-demand media asset as well as primary content thatis a live broadcast.

When the primary content is an on-demand video asset steps in blocks201-203 may be applied to obtain the average emotional arc. When theprimary content is an on-demand video asset, to display the primarycontent, the system may access an on-demand database that includes aplurality of videos and receive a video stream that includes the primarycontent. The primary content may be displayed to a plurality of users,and, during its display, biometric data from the users' display devicesis collected. Such biometric data is collected, such as on aframe-by-frame or scene-by-scene basis, while the primary content isbeing displayed. The various emotional states experienced during theconsuming of the primary content are represented using Russell'scircumplex model of emotions as mentioned above. Since each user'semotional response differs based on their personal feelings during thedisplay of scenes in the primary content, each emotional arc generatedfor a user differs from that of another user. The system then calculatesan average of all the users' emotional arcs to determine an averageemotional arc.

The amount and type of emotional historical data gathered for emotionalresponses from users that have consumed primary content or at least asegment of the primary content may differ based on the type of primarycontent. For example, the amount and type of historical emotional datafor primary content that is a live broadcast differs from the amount andtype of emotional data if the primary content is an on-demand mediaasset. If the primary content is an on-demand media asset, then theemotional data, which is gathered through biometric scans and otherinput means, can be collected from a plurality of users that havepreviously consumed the on-demand primary content at any time prior toit being consumed by the current user. Such emotional data may begathered at different times whenever the prior users are consuming theon-demand primary media. For example, one prior user may have consumedthe primary content a day prior, another prior user may have consumedthe primary content a week prior, and yet another prior user severalmonths prior to the current date. The system may store all suchemotional data and generate an average arc based on the emotional datacollected over the lifetime of the primary content, over a past year,past month, or any defined period. If such data is available much priorto a current user consuming the primary content, the system will haveadequate time and adequate data points to use such emotional data fromprior users in estimating what emotional response to expect from acurrent user consuming the same primary content.

The prior emotional data for an on-demand primary media asset may beavailable for segments not yet consumed by the current user. Forexample, if the user is currently consuming the on-demand primarycontent and is current at a play position that is 20 minutes into amovie, the prior data may be available for scenes that are later in theprimary content, such as scenes that will be at 30-, 40-, or 50-minutetime markers in the primary content. Having such data available forupcoming segments of the primary content provides the system theforesight needed in estimating how the current user may react to anupcoming scene based on how most users reacted to such scene. Such datamay be used in determining a type of secondary content to insert thatwill evoke a desired response when the primary content has evoked theestimated response in the current user.

In a different embodiment, when the primary content is a live broadcast,then the system will not have emotional data on upcoming scenes since noone has consumed scenes that have not yet been displayed. In thisembodiment, the system may determine average emotional responses frompreviously consumed segments of the live broadcast and generate aprojected emotional arc based on the consumed segments.

Referring back to FIG. 1 , at block 102, the average emotional arcgenerated at block 101 is used as an estimate to predict likely emotionsof the current user for the same primary content. In other words, thecurrent user's emotional arc for the primary content is estimated basedon the average emotional arc, also referred to as the default emotionalarc, of the primary content, with offsets for how the current user'sactual emotional arc, which includes values of activation and valence,differs during the consumption from the average activation and valenceof the default emotional arc of previous viewers/users at the same pointin the video. Accordingly, the system makes a prediction, based on theaverage emotional arc, of the likely emotional responses to expect fromthe current user when the current user is watching the primary content,i.e., the same as those in the average emotional arc or within apredetermined or threshold offset or range of the average emotional arc.The offsets and ranges may be defined by the system, the creator of theprimary or secondary content, the network, the broadcast channel, or anyentity affiliated with the service, production, or creation of theprimary and secondary content. For example, if an average emotional arcvalue at a particular play position in the primary media asset is at a“73,” the system may preset a range of +/−5 as an acceptable offset orrange within which an actual emotional response if considered to be amatch with the average emotional arc.

If the primary content is an on-demand media asset, then data forpreviously consumed segments of the primary content may be available forsegments that are not yet consumed by the current user. As such,emotional arc values, i.e., activation and valence values, for anupcoming scene or time marker in the primary content may also be knownbased on the previously consumed data from other users, such as, forexample, a scene relating to a car chase at a time marker 15:20 in amovie. Using this data, an estimate can be made on the activation andvalence values to expect when the current user also watches the carchase that is occurring at a time marker 15:20 in the movie.

In one embodiment, a predetermined insertion point in the primarycontent may be designated at time marker 15:21, i.e., immediatelyfollowing or perhaps still during the car chase scene in progress at15:20. Based on the estimation made earlier of prior users' emotionalstates at time marker 15:20, secondary content that will likely have thehighest impact based on the current user's emotional state is insertedinto the video stream and displayed beginning at time marker 15:21.

To determine what type of secondary content will have the highestimpact, the system analyzes a plurality of factors. These factorsinclude estimated activation and valence values of the current user atthe time marker 15:20, i.e., immediately preceding the time perioddesignated for inserting the secondary content. Understanding the user'smood and emotional state immediately prior to the insertion point of thesecondary content or secondary content allows the system to insertsecondary content that utilizes the emotional state of the user andpresents content that would further evoke the same or similar emotions,enhance the emotional state, or calm the emotional state, as desired. Byutilizing the already-generated emotional state, the system aims toincrease the user's attention and engagement with the secondary contenthaving correlation with the generated emotional state. For example,displaying secondary content that correlates with the generatedemotional state increases the likelihood of producing a higher impactand level of engagement with the secondary content. If such emotionalcorrelation is not taken into account, then a non-targeted secondarycontent that is displayed at this insertion point may not relate to theemotional content of the primary content and may be at odds with theuser's emotional state formed in response to the primary content.

For example, as mentioned above, in one embodiment, a scene displayedimmediately preceding the time period designated for inserting thesecondary content, such as immediately before the commercial break, maybe a car chase. Based on the average emotional arc, the system maydetermine the emotional state of the user is likely one of excitement,fun, and high energy. Based on this knowledge, the system may insertsecondary content showing a car for sale in which the secondary contentdisplays the car being driven fast and taking sharp turns, i.e., evokingfeelings of high energy, fun, and excitement. Inserting such a secondarycontent that is aligned with the user's emotional state may furtherenhance that state and lead to a higher level of engagement with thesecondary content.

In addition to valence and activation, other factors analyzed mayinclude trajectory of the emotional arc, acceleration or deceleration ofthe emotional arcs, such as rate of change, evoked by the primarycontent and the secondary content, content of the primary content at thepoint where the secondary content is to be inserted, and whether thecurrent user's emotional arc is within a range of the average emotionalarc.

Referring to block 103, if the values of the average emotional arc matchthe values of the current user's emotional arc, as shown in scenario 1,then, at block 104A, secondary content that is predetermined based onthe values of the average arc is inserted. For example, as depicted inblock 103, at play position 2, the value of the average emotional arc is13 and the current user's emotional arc is also 13. Since the values arethe same, the system determines that the actual user's response is inline with the average or expected emotional response and insertssecondary content that has been predetermined to display if an emotionalvalue of 13 is met. A match between average and current emotional arcwould also have been confirmed if the current user's actual emotionalarc value was 12 or 14 at play position 2 if the system allowed anoffset or range of +/−1 to be within the range. Any other offset orrange value or values within a predetermined threshold may also be usedto consider a match.

As described above, the system determines the actual user's response andcompares it with the average or expected emotional response. To do so,the system collects the emotional data used to determine the averageemotion arc and the current user's arc using similar methodologies tohelp make the data more consistent and accurate.

In another embodiment, if the values of the average emotional arc, whichis used as an estimate for the current user, do not match the values ofthe current emotional arc, as shown in scenario 2, then the currentvalues of the user's emotional arc are determined. As shown in block103, in one embodiment the current value of the current user's emotionalarc may be “7.” Even if the system sets an allowed offset or range of+/−1 to be within the range of the average emotional arc, since “7” isfar below the average emotional arc value of “13” at play position 2 inscenario 2, no match is confirmed. As such the system moves to block104B and secondary content is designated for an associated emotional arcvalue, i.e., an emotional arc value of “7,” if available, is inserted.If secondary content for an emotional arc value is not available, sinceit was not created or available in a database, then secondary contentdesignated for a value closest to the determine current emotional arcvalue of “7” will be inserted.

Referring back to block 103, in one embodiment, the system may use theuser's actual emotional arc, which is based on the user's currentvalence and activation score as a determination factor for determiningwhat type of secondary content to be inserted. In one embodiment thesystem may look at any of the combinations of the average emotional arcscores and running emotional arc score of the user, for example, thesystem may look at 1) only the average emotional arc score, 2) only theuser's running emotional score, or 3) a combination of both the averageemotional arc score and the user's running emotional arc score. In yetanother embodiment, the system may look at the average emotional scoreif a first threshold is met and if the threshold is exceeded then it maylook at the running emotional arc score. The threshold may be set by thesystem and dynamically changed for different media assets. The firstthreshold may be set at a number that is in close proximity to theaverage emotional score, such as +/−2 for an emotional a certation playposition and if that is exceeded then the user's running average scoremay be used. The close proximity may be a determining factor that theuser's emotional state is within a close proximity to what was estimatedas the emotional score at a certain play position.

FIG. 3 is a block diagram of an example process for gauging a user'semotional response to a first part of a secondary content and adjustinga second part of the secondary content based on the user's emotionalresponse, in accordance with some embodiments of the disclosure.

At block 301, a first part of a multiple-part secondary content isdisplayed. The multiple-part video may be a multiple-part secondarycontent, such as those shown during a Superbowl, where the contentrelates to the same product, but the messaging is split into two or moredifferent sequential pieces of secondary contents that carry through astoryline.

At block 302, the user's emotional response to the first part of thesecondary content may be determined using collected biometric data, andan emotional arc for the first part of the secondary content may begenerated. The biometric data may be collected on a frame-by-frame or asegment-by-segment basis, on a periodic basis, or at certain key pointsin the primary content. The biometric data, such as heart rate, gaze andgalvanic skin response may be collected via the display device's camera,heart rate monitor, sensors, and other components. The collectedbiometric data is indicative of the user's emotions in response toconsuming the displayed primary content. The various emotional statesexperienced during the consumption of the primary content may berepresented using Russell's circumplex model of emotions as discussedherein.

At block 303, the user's emotional response to the display of the firstpart of the secondary content may be compared with an average response,as mentioned earlier also referred to an estimated response. In oneembodiment, the secondary content may have been previously shown to aplurality of users. Based on previous consumption of the secondarycontent, the system may collect statistics of the emotional impactexperienced by those secondary content users who paid attention to thesecondary content. These statistics, which may be obtained throughbiometric scans as mentioned above, include activation and valencevalues in connection with the users' emotional response to consuming thesecondary content. The statistics may be used in generating an emotionalarc for each user. A calculation may be made based on all the emotionalarcs for all the users that have consumed the first part of thesecondary content to generate an average emotional arc for the firstpart of the secondary content. The values of the emotional arc then usedas an estimate of what type of emotional experience may be expected fromthe current user, who may be watching the first part of the secondarycontent for the first time.

As depicted in block 303, if the current user's emotional arc valuesmatch the average emotional arc values, then a match is confirmed. Anyother offset or range value or values within a predetermined thresholdmay also be used to consider a match. The comparison may be performed ona scene-by-scene, frame-by-frame, or a segment-by-segment basis. It mayalso be performed overall for the entire secondary content.

As depicted in scenario 1, at play position 1 in the secondary content,the estimated value is “8” for an average emotional arc that wasgenerated based on emotional values of users that have previouslyconsumed the first part of the secondary content. The confirming of amatch indicates that the actual emotional response of the user is thesame or within an accepted range of the estimated emotional response.For example, the estimated emotional response value is “7” and if theacceptable range is a +/−1 then the response of “8” is considered to bea match. The confirming of the match also means that the emotionalimpact desired by the producer of the first part of the secondarycontent was matched with the actual emotional biometric data, or thegenerated emotional arc, of the current user when consuming the firstpart of the secondary content. When the match is made, the second partof the secondary content that was designated for a match is played.

As depicted in scenario 2, at play position 2 in the first part of thesecondary content, the estimated value is “13” for an average emotionalarc that was generated based on emotional values of users that havepreviously consumed the first part of the secondary content. Since theaction value based on the current user's consumption of the first partof the secondary content is “7,” which is far below the estimated value,the system may determine that a match is not made. Not having a matchindicates that the actual emotional response of the user is not the sameor within an accepted range of the estimated emotional response for thefirst part of the secondary content. No match also means that theemotional impact desired by the producer of the first part of thesecondary content was not matched with the actual emotional biometricdata, or the generated emotional arc, of the current user and insteadthe user had a different emotional response. When the match is not made,the second part of the secondary content that was designated for a matchis not played, and instead, the system obtains a variation of the secondpart of the secondary content that matches the emotional arc value “7”of the user, if such a variation is already prepared and is availablefor playing.

In one embodiment, the secondary content may be shown in real time toall the users and may not have been shown previously to anyone. In suchembodiments where all the users are shown the secondary content in realtime, including the current user, biometric data for any priorconsumption may not be available. As such, the system may collect thebiometric data of the current user for the first part of the secondarycontent to determine its impact on the current user. The collectedbiometric data may be used to generate an emotional arc. Since theproducer, creator, or provider of the secondary content may have adesired impact for the first part of the multiple-part secondarycontent, the user's collected biometric data for the first part may becompared with the desired emotional arc for the first part of thesecondary content.

In one embodiment, if the emotional response of the user for the firstpart of the secondary content matches the desired emotional response,such as in scenario 1 at block 303, then the second part of thesecondary content which was designated based on there being a match isplayed. In another embodiment, if the actual emotional response does notmatch the desired emotional response, such as in scenario 2 in block303, then steps described in block 304B are executed.

FIG. 4 is a table of options for analyzing emotional responses relatingto the primary content, secondary content, and the current user fordetermining impact of secondary content, in accordance with someembodiments of the disclosure.

In another embodiment, as depicted in “Option A,” emotional responses tothe primary content and real-time emotional responses from the currentuser may be analyzed to determine when and what type of secondary assetto insert in the primary content.

In this embodiment, the following options may be provided for insertingthe secondary content into the primary content: a) flexibility to insertthe secondary content anywhere in the primary content, b) restriction toinsert the secondary content during a designated commercial break, andc) restriction to insert the secondary content during a commercial breakthat is taken depending upon the activity or stage of the primarycontent, such as occurrence of a commercial break whenever a time out istaken in a game.

In one embodiment, if the primary content is an on-demand media asset,then the primary content may be displayed to a plurality of users andtheir emotional responses collected to generate an average emotionalprimary arc. The system may then monitor the current user's emotionalresponses in real time to the displayed primary content, and, if theuser's emotional arc values intersect with the average emotional primaryarc, then a predetermined secondary content that has been designated forthe emotional arc values may be displayed to the current user.

When there is flexibility to insert the secondary content anywhere inthe primary content, then the system may monitor the emotional responsesof the current user in real time and compare them to the averageemotional arc. Whenever the emotional response is aligned with theaverage emotional arc or aligned within a range of the average emotionalarc, the system may determine to insert the secondary content.

When the insertion of the secondary content is restricted, either basedon a scheduled commercial break that is to occur at a designated time,such as 11:45 AM, or based on whenever a time out is taken in a game,the system may track the most recent emotional response of the currentuser to the primary content and use that to determine whether to insertthe secondary content. For example, the system may determine that theemotional arc that is generated based on segments of the primary contentalready consumed is the emotional response desired by the maker of thesecondary content as an insertion point of the secondary content intothe primary content. This may be because the secondary content includescontent that would likely continue the emotional response that wasevoked by the current user and having such continuity in emotionalresponse produces a higher impact for the secondary content. The makerof the secondary content may also look for a particular range ofemotional arc values that would make the content of the secondarycontent more impactful, such an activation and valence value that isaligned with a certain level of excitement, to play secondary contentthat continues that feeling of excitement.

In another embodiment, if the primary content is a live broadcast, thenthe system may track the user's emotional responses to the livebroadcast in real time and insert a designated secondary content thatmeets an emotional response value of the user.

In another embodiment, as depicted in “Option B,” emotional responses tothe secondary content and from a plurality of users that have previouslyconsumed the secondary content and the real-time emotional responsesfrom the current user as they watch the first part of the secondarycontent may be analyzed to determine when and what type or version of asecond part of the secondary asset to insert in the primary content.

In this embodiment, secondary content may contain multiple parts where afirst part is displayed during a first commercial break and a secondpart is displayed at a subsequent commercial break. For example, duringevents such as Super Bowl, Olympics, NBA games, which have millions ofviewers tuned in to watch the live broadcast, secondary content havingmultiple parts is often displayed. Typically, the multiple-partsecondary content, which is either a two-part or a three-part secondarycontent, is created by the same advertiser and is focused on a singleproduct. Such multiple-part secondary content has a continuity ofmessage from a first part to a second part (and possibly a third part).

In this embodiment, Option B, where the secondary content is split intomultiple parts, the system determines the desired impact, i.e.,emotional response, of the first part of the secondary content andmeasures it against the actual emotional response of the user consumingthe first part of the secondary content. The process of determining theactual emotional response includes obtaining the user's biometric datathrough a plurality of means. As mentioned earlier, these includetracking the user using a camera of the media device displaying thesecondary content, tracking the user's heart rate, motions, speech, andany other verbal or written content produced by the user in relation tothe secondary content being consumed. The system may track thatemotional response during the progress of the secondary content on ascene-by-scene or a segment-by-segment basis. These emotional responsesinclude values of valence and activation, and such values, along withany other factors, may be used in generating an emotional arc of theuser that correlates with the progress of display from start to end ofthe secondary content.

If it is a live broadcast, then the system may obtain the desiredemotional response that is targeted by the creator of the secondarycontent. If it is an on-demand media asset where the secondary contenthas been shown to a plurality of users previously, then the system mayobtain an average emotional arc that is generated based on priorconsumption of the secondary content.

Whether it is the desired emotional response for a piece of secondarycontent that is being shown for the first time or it is an averageemotional response of a previously consumed secondary content, thesystem may then compare either response to the current, actual responsein real time from the current user.

If the user's response to the first part of the secondary content meetsthe desired emotional response, or the estimated emotional response,then the advertiser may continue to show the second part of thesecondary content at a subsequent commercial break that has beendesignated based on the user's response meeting the desired or expectedemotional response. This means that the type of emotional response thatthe creator of the secondary content desires to evoke with a two-partsecondary content is on track. Because the first part of the secondarycontent matched with the desired or expected emotional response, thereis a likelihood that the second part of the secondary content will alsomeet the desired or expected emotional response. On the other hand, insome cases, the system determines that the first part of the secondarycontent did not receive an emotional response that was desired or theaverage emotional response that was expected. Then the system maydetermine the values of the emotional response received based on theuser's consumption and display a different version of the second part ofthe secondary content that matches the values of the emotional responsereceived. In this scenario, the advertiser may have previously created afew variations of the secondary content that meet different ranges ofemotional responses and have them ready to be displayed based on thevalues obtained of actual emotional response for a user that hasconsumed the first part of the secondary content.

In another embodiment, as depicted in “Option C,” emotional responsesmay be determined for a multi-part secondary content as follows: 1)determine average emotional response to primary content and 2) insertthe first part of the secondary content at a play position in theprimary content where the average emotional response is as desired, 3)determine the real-time emotional response of the user to the insertedfirst part of the secondary content, 4) if the real-time emotionalresponse of the user to the inserted first part is as desired, theninsert the second part of the secondary content as predetermined, 5) ifreal-time emotional response of the user to the inserted first part isnot as desired, determine actual value of emotional response and inserta second part of the secondary content that matches the actual value ofthe emotional response. These steps are further described below fordifferent variations of flexibility allowed in where to insert thesecondary content in the primary content, i.e., at predetermined pointsor whenever desired.

In one embodiment, an average emotional arc may be generated for primarycontent. The average emotional arc may be generated based on displayingthe primary content to a plurality of users and obtaining theiremotional responses as the primary content progresses fromscene-to-scene, frame-to-frame, or segment-to-segment. An averageemotional arc based on the collective emotional arcs of all the userswho have previously consumed the primary content may be generated. Thevalues of the average emotional arc may be displayed as data in a tableor as a graph or any other desired format.

Based on the values of the average emotional arc, in one embodiment, ifthe system allows flexibility to insert secondary content at any stagein their primary content, then secondary content that is created for atargeted emotional response having a targeted emotional arc value, suchas a specific rage of activation and valence values, may be displayedwhen the emotional arc values for the primary content reach the targetedemotional arc value.

In another embodiment, the system may not allow flexibility to insertsecondary content at any stage in their primary content. The system mayallow insertion of the secondary content only during a designatedcommercial break or during a commercial break that is taken dependingupon the activity or stage of the primary content, such as occurrence ofa commercial break whenever a timeout is taken in a game. In suchembodiments, under Option C, the system may track the real-timeemotional response of the user and obtain values of their emotional arc.If the last data point of the emotional response prior to the commercialbreak matches the average emotional arc calculated based on other user'semotional responses for the primary content, then the system may insertthe secondary content based on values of the last emotional response.Any other offset or range value or values within a predeterminedthreshold may also be used to consider a match.

In yet another embodiment, at the first stage, under Option D, thesystem may the determine the average emotional arc for the primarycontent, based on prior consumption data, and insert a first part of thesecondary content that meets a desired arc value in the averageemotional arc. Next, the system may determine, in real time, the actualemotional response of the current user with respect to the first part ofthe secondary content. If the actual emotional response meets a desiredemotional response that is targeted for the first part of the video,then the system may insert the second part of the secondary content aspreviously designated.

In yet another embodiment, as depicted in Option D, the primary contentmay be displayed to two separate groups of pluralities of users thathave different viewpoints or interest. For example, the primary contentmay be a game between two NBA™ basketball teams, the Golden StateWarriors™ and the Chicago Bulls™. In this embodiment, a first pluralityof users may be selected as users that are fans of the Chicago Bulls andits second plurality of users may be selected as users that are fans ofthe Golden State Warriors. A determination of the user, or the group ofusers, aligned interest with a team may be obtained based on theirprofiles, prior consumption history, their statements made that may becaptured by a microphone, their subscription to certain teams, theirpurchases of team merchandise that can be determined based on theironline purchase history, or any other methods that can be used todetermine their alignment with a team, such as either Chicago Bulls orGolden State Warriors. In a game between the two teams, depending onwhich team is winning, the emotional status of the first plurality ofusers may differ from the emotional status of the second plurality ofusers. The emotional status may also be similar between the twopluralities of users when the teams scores are in close proximity ofeach other creating a stressful emotional status in fans of both teams.

In this scenario, the system may obtain biometric data for the fans ofthe Chicago Bulls and separately for the fans of the Golden StateWarriors. The system may generate an average emotional arc for all theChicago Bulls fans and another emotional art for all the Golden StateWarrior fans. The system may then determine the current user's responsein real-time to determine whether the user is aligned with the ChicagoBulls or the Golden State Warriors. Depending on their alignment, therespective average emotional arc may be used. For example, based on theuser's current emotional response, if the system determines that theuser is aligned with the Golden State Warriors, then the system maycompare the users emotional arc with the average emotional arc generatedfor the Golden State Warrior fans. If the current user's emotional archad a play position in the game is same or within a range of values ofthe average emotional arc generated for the Golden State Warrior fans,then the system may insert secondary content that has been targeted forthe value of the average emotional arc. Although a sports example hasbeen used to illustrate how opposing views, different views, differentinterest, and different affiliations can be captured through obtainingdata and grouping users based on their interests and affiliations, theembodiments are not so limited. For example, differing views, interests,for any work, business, or general day-to-day life matters wheredifferent interest and affiliations exist can also be captured throughthe described embodiments.

Although some combinations of displaying a video to a first and/or asecond plurality of users, along with various combinations or generatingaverage emotional arcs and current emotional arc, and comparison betweenthe emotional arcs, are described, the embodiments are not so limitedand other combinations are also contemplated.

FIG. 5 is a block diagram of various components of a system fordetermining emotional arcs and inserting secondary content, inaccordance with some embodiments of the disclosure. In one embodiment, amedia device associated with a current user, media devices associatedwith a plurality of users, a video stream and secondary content learner,and a video stream and secondary content server may be used for theprocesses described in FIGS. 1-4 and 6 .

In one embodiment, primary content is sent from a video stream andsecondary content learner or device 510 to other user's media device505, i.e., not the current user, via video stream 503. The primarycontent may be sent to the user's media device 505 in response to theother user's media device(s) 505 requesting the primary content forconsumption from the video stream and secondary content learner ordevice 510. The primary content may be an on-demand media asset or alive broadcast.

In one embodiment, the media device 505 may include a display and aplurality of sensors and biometric scanning devices. For example, thedevice 505 may include and/or otherwise be operatively orcommunicatively coupled to a camera, a motion detector, a gyroscope, andan accelerometer, and biometric devices such as a heart rate monitor,and a pulse or skin sensor. The device 505 may also include a processorconnected to control circuitry, such as displayed in an example deviceof FIG. 8 , for executing instructions to perform the functions relatingto obtaining the viewer's emotional data. For example, the controlcircuitry may activate the camera associated with the device 505 totrack the viewer's gaze and determine if the viewer's gaze is focused atthe displayed content in the primary content. The control circuitry mayalso access the gyroscope and accelerometer there to determine theviewers' motions in response to watching the primary content. Thecontrol circuitry may also monitor the other user's heartbeat todetermine the other user's emotions. In addition to the above-mentionedcomponents of the media device, other components and sensors associatedwith the other user's media device may also be used by control circuitryto monitor the other user's emotional responses throughout the course ofthe display of the primary content from its start to end.

In one embodiment, the other user's gaze and other biometric datarelating to the other user's emotional response to consuming variousframes, scenes, and segments of the primary content obtained by themedia device 505 may be sent to the video stream and secondary contentlearner service or device 510 as depicted at 507.

The video stream and secondary content learner service or device 510receives the gaze and biometric data of each other user, from aplurality of other users, and generates an emotional arc for the otheruser. The video stream and secondary content learner service or device510 also calculates average activation and valence values based on thegaze and biometric data received from each other user to generate anaverage that represents all the plurality of other users. The emotionalarc includes the average activation and valence values of the pluralityof other users and in some embodiments, it includes activating, valence,and other factors that relate to the emotional response by the pluralityof other users that have consumed the primary content. The video streamand secondary content learner service or device 510 transmits thegenerated average emotional arc to the Video stream and secondarycontent server 520.

The video stream and secondary content server 520 receives at 514 theaverage emotional arc generated on the basis of emotional data from theplurality of other users that have consumed the primary content and alsoreceives at 522 emotional response data (e.g., gaze and biometric data)from a current user's media device 530. The video stream and secondarycontent server 520 generates an emotional arc for the current user basedon the received gaze and biometric data, for instance. The video streamand secondary content server 520 then compares the average emotional arcwith the current emotional arc of the current viewer.

As discussed above, the average emotional arc and the current emotionalarc reflect emotional responses of respective user(s) over a course oftime of consumption or playback of the primary content. Using the dataof both arcs, the video stream and secondary content server 520determines intersecting points between the average emotional arc and thecurrent emotional arc, where points of intersection are combined valuesof each respective arc. The determination comprises determining playpositions in the primary content where both arcs have the same values orvalues within a threshold range of each other. The video stream andsecondary content server 520 may then insert secondary content where thevalues are within the threshold range of each other.

Likewise, a first part of secondary content may be sent from the videostream and secondary content learner or device 510 to the user's mediadevice 505 as depicted at 503. Similar processes as mentioned above forthe primary content may be performed for the first part of the secondarycontent to generate an average emotional arc on the basis of emotionaldata from the plurality of users that have consumed the first part ofthe secondary content and also a current emotional arc for the currentuser consuming the first part of the secondary content. The video streamand secondary content server 520 may then compare the average emotionalarc with the current emotional arc of the current user for the firstpart of the secondary content. If the values of both arcs are the sameor within a range, such as a predetermined threshold range, of eachother, then the video stream and secondary content server 520 may inserta second part of the secondary content as designated for the matchedvalue. If the values do not match or are not within the threshold rangeof each other, then a second part of the secondary content, ifavailable, for the value determined at the play position as the currentemotional arc value will be inserted. Although only certain combinationshave been described in reference to FIG. 5 , other combinations asdescribed in FIG. 4 may also be implemented using the components of FIG.5 .

FIG. 6 is a block diagram of communications between different componentsof the system for determining emotional arcs and inserting secondarycontent, in accordance with some embodiments of the disclosure. Thecommunications described in this figure relate to actions performed bythe components of the system in FIG. 5 . For example, some of theseactions performed include executing processes described in FIGS. 1-4 .

As depicted in FIG. 6 , in some embodiments, a primary content istransmitted from the video stream and secondary content learner, such asthe video stream and secondary content learner 510 depicted in FIG. 5 ,to a group of users (also referred to as other users) on their mediadevices, such as media device 505 depicted in FIG. 5 . The primarycontent, in one embodiment, is sent to the others users to survey andget their emotional response. The emotional responses from thetransmitted primary content are collected and transmitted to the videostream and secondary content server, such as the video stream andsecondary content server 520 in FIG. 5 .

Likewise, a first part of a secondary content, such as secondary contenthaving multiple parts as described in the description related to FIG. 3above, may also be transmitted from the video stream and secondarycontent learner 510 to media devices 505 of other users to obtain theiremotional responses. The first part of the secondary content may also besent to the current user to get their emotional response.

The video stream and secondary content server 520 may perform analysisof the emotional data received and generate an average emotional arcthat can be used as an estimate for the current user.

The video stream and secondary content server 520 may then transmit theprimary content to the current viewer and gather the current viewer'sactual emotional response. The actual emotional response, which may beused to generate a user's emotional arc, may then be compared with theaverage emotional arc to determine if the estimates were accurate. Ifthere is a variance with the estimate, such as due to the current user'semotional response being different to the primary content than the groupof users surveyed previously or due to the current user having consumedthe primary content multiple times, then an adjustment or offset to theaverage emotional arc based on the amount of variance is made.

Once a determination is made that the current user's emotional arc iswithin a threshold of the average emotional arc, or adjustments andoffsets are made to bring them both within a threshold variance, thenthe video stream and secondary content server 520 may splice the videostream, as depicted in 524 of FIG. 5 . The video stream and secondarycontent server 520 may insert the secondary content into the splicedvideo stream and transmit it to the current user's media device, such asmedia device 530 depicted in FIG. 5 .

FIG. 7 is a block diagram of an example system for determining emotionalresponses for videos, generating associated emotional arc, anddetermining insertions of secondary content, in accordance with someembodiments of the disclosure. In FIG. 7 , an audio/video system 700 isconfigured, in accordance with some embodiments of the disclosure. Insome embodiment, one or more parts of, or the entirety of system 700,may be configured as a system implementing various features, processes,functionalities, tables, emotional scores, activation and valencevalues, user interfaces, and components for processing functionalitydescribed in FIGS. 1-4 and 6 . Although FIG. 7 shows a certain number ofcomponents, in various examples, system 700 may include fewer than theillustrated number of components and/or multiples of one or more of theillustrated number of components.

System 700 is shown to include a computing device 718, a server 702 anda communication network 714. It is understood that while a singleinstance of a component may be shown and described relative to FIG. 7 ,additional instances of the component may be employed. For example,server 702 may include, or may be incorporated in, more than one server.Similarly, communication network 714 may include, or may be incorporatedin, more than one communication network. Server 702 is showncommunicatively coupled to computing device 718 through communicationnetwork 714. While not shown in FIG. 7 , server 702 may be directlycommunicatively coupled to computing device 718, for example, in asystem absent or bypassing communication network 714.

Communication network 714 may comprise one or more network systems, suchas, without limitation, an Internet, LAN, WIFI or other network systemssuitable for audio processing applications. In some embodiments, system700 excludes server 702, and functionality that would otherwise beimplemented by server 702 is instead implemented by other components ofsystem 700, such as one or more components of communication network 714.In still other embodiments, server 702 works in conjunction with one ormore components of communication network 714 to implement certainfunctionality described herein in a distributed or cooperative manner.Similarly, in some embodiments, system 700 excludes computing device718, and functionality that would otherwise be implemented by computingdevice 718 is instead implemented by other components of system 700,such as one or more components of communication network 714 or server702 or a combination. In still other embodiments, computing device 718works in conjunction with one or more components of communicationnetwork 714 or server 702 to implement certain functionality describedherein in a distributed or cooperative manner.

Computing device 718 includes control circuitry 728, display 734 andinput circuitry 716. Control circuitry 728 in turn includes transceivercircuitry 762, storage 738, which receives input from transceivercircuitry 762 via 752, and processing circuitry 740. In someembodiments, computing device 718 or control circuitry 728 may beconfigured as media device 505 or 530 of FIG. 5 .

Server 702 includes control circuitry 720 and storage 724. Each ofstorages 724 and 738 may be an electronic storage device. As referred toherein, the phrase “electronic storage device” or “storage device”should be understood to mean any device for storing electronic data,computer software, or firmware, such as random-access memory, read-onlymemory, hard drives, optical drives, digital video disc (DVD) recorders,compact disc (CD) recorders, BLU-RAY disc (BD) recorders, BLU-RAY 3Ddisc recorders, digital video recorders (DVRs, sometimes called personalvideo recorders, or PVRs), solid state devices, quantum storage devices,gaming consoles, gaming media, or any other suitable fixed or removablestorage devices, and/or any combination of the same. Each storage 724,738 may be used to store various types of content, metadata, and orother types of data (e.g., they can be used to store activation andvalence values, emotional arc values, and biometric data from users).Nonvolatile memory may also be used (e.g., to launch a boot-up routineand other instructions). Cloud-based storage may be used to supplementstorages 724, 738 or instead of storages 724, 738. In some embodiments,the audio and/or video portion(s) of the breakout room session may berecorded and stored in one or more of storages 712, 738.

In some embodiments, control circuitry 720 and/or 728 executesinstructions for an application stored in memory (e.g., storage 724and/or storage 738). Specifically, control circuitry 720 and/or 728 maybe instructed by the application to perform the functions discussedherein. In some implementations, any action performed by controlcircuitry 720 and/or 728 may be based on instructions received from theapplication. For example, the application may be implemented as softwareor a set of executable instructions that may be stored in storage 724and/or 738 and executed by control circuitry 720 and/or 728. In someembodiments, the application may be a client/server application whereonly a client application resides on computing device 718, and a serverapplication resides on server 702.

The application may be implemented using any suitable architecture. Forexample, it may be a stand-alone application wholly implemented oncomputing device 718. In such an approach, instructions for theapplication are stored locally (e.g., in storage 738), and data for useby the application is downloaded on a periodic basis (e.g., from anout-of-band feed, from an Internet resource, or using another suitableapproach). Control circuitry 728 may retrieve instructions for theapplication from storage 738 and process the instructions to perform thefunctionality described herein. Based on the processed instructions,control circuitry 728 may determine a type of action to perform inresponse to input received from input circuitry 716 or fromcommunication network 714. For example, in response to determining thata current emotional arc value meets the average emotional arc value, thecontrol circuitry 728 may perform the steps of processes described inFIGS. 1-3 to insert secondary content associated with the average arcvalue.

In client/server-based embodiments, control circuitry 728 may includecommunication circuitry suitable for communicating with an applicationserver (e.g., server 702) or other networks or servers. The instructionsfor carrying out the functionality described herein may be stored on theapplication server. Communication circuitry may include a cable modem,an Ethernet card, or a wireless modem for communication with otherequipment, or any other suitable communication circuitry. Suchcommunication may involve the Internet or any other suitablecommunication networks or paths (e.g., communication network 714). Inanother example of a client/server-based application, control circuitry728 runs a web browser that interprets web pages provided by a remoteserver (e.g., server 702). For example, the remote server may store theinstructions for the application in a storage device. The remote servermay process the stored instructions using circuitry (e.g., controlcircuitry 728) and/or generate displays. Computing device 718 mayreceive the displays generated by the remote server and may display thecontent of the displays locally via display 734. This way, theprocessing of the instructions is performed remotely (e.g., by server702) while the resulting displays, such as the display windows describedelsewhere herein, are provided locally on computing device 718.Computing device 718 may receive inputs from the user via inputcircuitry 716 and transmit those inputs to the remote server forprocessing and generating the corresponding displays. Alternatively,computing device 718 may receive inputs from the user via inputcircuitry 716 and process and display the received inputs locally, bycontrol circuitry 728 and display 734, respectively.

Server 702 and computing device 718 may transmit and receive content anddata such as biometric data from user that have consumed a primary orsecondary content, data related to a user's gaze when consuming a video,emotional arc data, and data relating to an emotional impact thresholdas desired by the creator of the secondary content, and video and mediacontent via communication network 714. For example, server 702 may be asecondary content provider, and computing device 718 may be a mediadevice configured to display the secondary content provided as depictedin FIGS. 5 and 6 . Control circuitry 720, 728 may send and receivecommands, requests, and other suitable data via links 706, 708, and 710through communication network 714 using transceiver circuitry 760, 762,respectively. Control circuitry 720, 728 may communicate directly witheach other using transceiver circuits 760, 762, respectively, avoidingcommunication network 714.

It is understood that computing device 718 is not limited to theembodiments and methods shown and described herein. In nonlimitingexamples, computing device 718 may be a user's media device, a personalcomputer (PC), a laptop computer, a tablet computer, a WebTV box, apersonal computer television (PC/TV), a PC media server, a PC mediacenter, a handheld computer, a stationary telephone, a personal digitalassistant (PDA), a mobile telephone, a smartphone, or any other device,computing equipment, or wireless device, and/or combination of the samecapable of suitably collecting biometric and gaze data, displayingprimary and secondary content, and displaying emotional arcs.

Control circuitry 720 and/or 718 may be based on any suitable processingcircuitry such as processing circuitry 726 and/or 740, respectively. Asreferred to herein, processing circuitry should be understood to meancircuitry based on one or more microprocessors, microcontrollers,digital signal processors, programmable logic devices,field-programmable gate arrays (FPGAs), application-specific integratedcircuits (ASICs), etc., and may include a multi-core processor (e.g.,dual-core, quad-core, hexa-core, or any suitable number of cores). Insome embodiments, processing circuitry may be distributed acrossmultiple separate processors, for example, multiple of the same type ofprocessors (e.g., two Intel Core i9 processors) or multiple differentprocessors (e.g., an Intel Core i7 processor and an Intel Core i9processor). In some embodiments, control circuitry 720 and/or controlcircuitry 718 are configured to collect display primary or secondarycontent, collect user's biometric and gaze data, generate emotionalarcs, calculate an average emotional arc, determine current emotionalvalues of a current user, generate current emotional arcs, compareaverage with current emotional arcs, determine a point of insertion ofsecondary content, determine which secondary content, such as asecondary content, to insert, access secondary content from variousdatabases and services, insert secondary content into primary content,and collect emotional data relating to a first part of secondary contentto make decision on when or which second part of secondary content toinsert in the primary content and perform various related processesdescribed and shown in connection with FIGS. 1-4, 6, 9A-D and 12.

Computing device 718 receives a user input 704 at input circuitry 716.For example, computing device 718 may receive a input like a user'semotional response to a scene, frame, or segment of a primary orsecondary content consumed, which may be received be sent to thecomputing device from a camera, heartbeat measuring components, ormotion detecting components that are associated with the user's mediadevice. In some embodiments, computing device 718 is a media deviceassociated with the user that collects gaze and biometric data anddisplayed primary and secondary content. It is understood that computingdevice 718 is not limited to the embodiments and methods shown anddescribed herein. In nonlimiting examples, computing device 718 may be apersonal computer (PC), a laptop computer, a tablet computer, a handheldcomputer, a stationary telephone, a personal digital assistant (PDA), amobile telephone, a smartphone, or any other type of media device orequipment, computing equipment, or wireless device, and/or combinationof the same.

User input 704 may be received from a user selection-capturing interfacethat is separate from device 718, such as a remote-control device,trackpad or any other suitable user movement sensitive or capturedevices, or as part of device 718, such as a touchscreen of display 734.Transmission of user input 704 to computing device 718 may beaccomplished using a wired connection, such as an audio cable, USBcable, ethernet cable or the like attached to a corresponding input portat a local device, or may be accomplished using a wireless connection,such as Bluetooth, WIFI, WiMAX, GSM, UTMS, CDMA, TDMA, 3G, 4G, 4G LTE,or any other suitable wireless transmission protocol. Input circuitry716 may comprise a physical input port such as a 3.5 mm audio jack, RCAaudio jack, USB port, ethernet port, or any other suitable connectionfor receiving audio over a wired connection or may comprise a wirelessreceiver configured to receive data via Bluetooth, WIFI, WiMAX, GSM,UTMS, CDMA, TDMA, 3G, 4G, 4G LTE, or other wireless transmissionprotocols.

Processing circuitry 740 may receive input 704 from input circuit 716via 744, display 734, via 742, storage 738, via 750, and transceivercircuitry 762 via 754. Processing circuitry 740 may convert or translatethe received user input 704 that may be in the form of voice input intoa microphone, or movement or gestures to digital signals. In someembodiments, input circuit 716 performs the translation to digitalsignals. In some embodiments, processing circuitry 740 (or processingcircuitry 726, as the case may be) carries out disclosed processes andmethods. Processing circuitry 740 may also provide input intotransceiver circuitry 762 via 746, storage 738, via 748. For example,processing circuitry 740 or processing circuitry 726 may performprocesses described in FIGS. 1-4, and 6 .

FIG. 8 is a block diagram of a user's media device, in accordance withsome embodiments of the disclosure. In an embodiment, the user's mediadevice 800, is the same media device 718 of FIG. 7 . The user's mediadevice 800 may receive content and data via input/output (I/O) path 802.The I/O path 802 may provide audio content (e.g., speech input from auser consuming a primary or secondary content) and data, such as gaze orbiometric data, to control circuitry 804, which includes processingcircuitry 806 and a storage 808. The control circuitry 804 may be usedto send and receive commands, requests, and other suitable data usingthe I/O path 802. The I/O path 802 may connect the control circuitry 804(and specifically the processing circuitry 806) to one or morecommunications paths. I/O functions may be provided by one or more ofthese communications paths but are shown as a single path in FIG. 8 toavoid overcomplicating the drawing.

The control circuitry 804 may be based on any suitable processingcircuitry such as the processing circuitry 806. As referred to herein,processing circuitry should be understood to mean circuitry based on oneor more microprocessors, microcontrollers, digital signal processors,programmable logic devices, field-programmable gate arrays (FPGAs),application-specific integrated circuits (ASICs), etc., and may includea multi-core processor (e.g., dual-core, quad-core, hexa-core, or anysuitable number of cores) or supercomputer. In some embodiments,processing circuitry may be distributed across multiple separateprocessors or processing units, for example, multiple of the same typeof processing units (e.g., two Intel Core i7 processors) or multipledifferent processors (e.g., an Intel Core i5 processor and an Intel Corei7 processor).

The methods and processes involved in determining topics of collectingdisplay primary or secondary content, collecting user's biometric andgaze data, generating emotional arcs, calculate an average emotionalarc, determining current emotional values of a current user, generatecurrent emotional arcs, comparing average with current emotional arcs,determining a point of insertion of secondary content, determining whichsecondary content to insert, accessing secondary content from variousdatabases and services, inserting secondary content into primarycontent, and collecting emotional data relating to a first part ofsecondary content to make decision on when or which second part ofsecondary content to insert in the primary content and perform variousrelated processes described and shown in connection with FIGS. 1-4, 6,9A-D and 12 as described herein can be at least partially implementedusing the control circuitry 804. In one embodiment, automatic refers to,performing the function without user intervention. The processes asdescribed herein may be implemented in or supported by any suitablesoftware, hardware, or combination thereof. They may also be implementedon servers, such as server 102 in FIG. 7 , on remote servers, or acrossboth.

In client-server-based embodiments, the control circuitry 804 mayinclude communications circuitry suitable for communicating with one ormore servers that may at least implement the storing of activation andvalence values, emotional arc values, and biometric data from users andat least implement processes to display primary or secondary content,collect user's biometric and gaze data, generate emotional arcs,calculate an average emotional arc, determine current emotional valuesof a current user, generate current emotional arcs, compare average withcurrent emotional arcs, determine a point of insertion of secondarycontent, determine which secondary content to insert, access secondarycontent from various databases and services, insert secondary contentinto primary content, and collect emotional data relating to a firstpart of secondary content to make decision on when or which second partof secondary content to insert in the primary content and performvarious related processes described and shown in connection with FIGS.1-4, 6, 9A-D and 12.

The instructions for carrying out the above-mentioned functionality maybe stored on the one or more servers. Communications circuitry mayinclude a cable modem, an integrated service digital network (ISDN)modem, a digital subscriber line (DSL) modem, a telephone modem,Ethernet card, or a wireless modem for communications with otherequipment, or any other suitable communications circuitry. Suchcommunications may involve the Internet or any other suitablecommunications networks or paths. In addition, communications circuitrymay include circuitry that enables peer-to-peer communication of usermedia devices or various servers, such as advertising servers.

Memory may be an electronic storage device provided as the storage 808that is part of the control circuitry 804. As referred to herein, thephrase “electronic storage device” or “storage device” should beunderstood to mean any device for storing electronic data, computersoftware, or firmware, such as random-access memory, read-only memory,hard drives, optical drives, digital video disc (DVD) recorders, compactdisc (CD) recorders, BLU-RAY disc (BD) recorders, BLU-RAY 3D discrecorders, digital video recorders (DVR, sometimes called a personalvideo recorder, or PVR), solid-state devices, quantum storage devices,gaming consoles, gaming media, or any other suitable fixed or removablestorage devices, and/or any combination of the same. The storage 808 maybe used to store various types of content described herein, such asvalence values, emotional arc values, and biometric data from users.Nonvolatile memory may also be used (e.g., to launch a boot-up routineand other instructions). Cloud-based storage, described in relation toFIG. 7 , may be used to supplement the storage 808 or instead of thestorage 808.

The control circuitry 804 may include audio generating circuitry andtuning circuitry, such as one or more analog tuners, audio generationcircuitry, filters or any other suitable tuning or audio circuits orcombinations of such circuits. The control circuitry 804 may alsoinclude scaler circuitry for upconverting and down converting contentinto the preferred output format of the media device 800. The controlcircuitry 804 may also include digital-to-analog converter circuitry andanalog-to-digital converter circuitry for converting between digital andanalog signals. The tuning and encoding circuitry may be used by theconferencing device 800 to receive and to display, to play, or to recordprimary or secondary content. The circuitry described herein, including,for example, the tuning, audio generating, encoding, decoding,encrypting, decrypting, scaler, and analog/digital circuitry, may beimplemented using software running on one or more general purpose orspecialized processors. If the storage 808 is provided as a separatedevice from the media device 800, the tuning and encoding circuitry(including multiple tuners) may be associated with the storage 808.

The user may utter an emotional response to the control circuitry 804,such as during consumption of a primary or secondary content, which arereceived by the microphone 816. The microphone 816 may be any microphone(or microphones) capable of detecting human speech. The microphone 816is connected to the processing circuitry 806 to transmit detected voiceinput or commands and other speech thereto for processing. In someembodiments, voice assistants (e.g., Siri, Alexa, Google Home andsimilar such voice assistants) receive and process the voice commandsand other speech.

The media device 800 may include an interface 810. The interface 810 maybe any suitable user interface, such as a remote control, mouse,trackball, keypad, keyboard, touch screen, touchpad, stylus input,joystick, or other user input interfaces. A display 812 may be providedas a stand-alone device or integrated with other elements of the mediadevice 800. For example, the display 812 may be a touchscreen ortouch-sensitive display. In such circumstances, the interface 810 may beintegrated with or combined with the microphone 816. When the interface810 is configured with screen, such a screen may be one or more of amonitor, a television, a liquid crystal display (LCD) for a mobiledevice, active-matrix display, cathode ray tube display, light-emittingdiode display, organic light-emitting diode display, quantum dotdisplay, or any other suitable equipment for displaying visual images.In some embodiments, the interface 810 may be HDTV-capable. In someembodiments, the display 812 may be a 3D display. The speaker (orspeakers) 814 may be provided as integrated with other elements of themedia device 800 or may be a stand-alone unit. In some embodiments, thedisplay 812 may be outputted through speaker 814.

The device 800 of FIG. 8 can be implemented in system 700 of FIG. 7 asmedia device 718, but any other type of media device suitable forimplementing processes to display primary or secondary content, collectuser's biometric and gaze data, generate emotional arcs, calculate anaverage emotional arc, determine current emotional values of a currentuser, generate current emotional arcs, compare average with currentemotional arcs, determine a point of insertion of secondary content,determine which secondary content to insert, access secondary contentfrom various databases and services, insert secondary content intoprimary content, and collect emotional data relating to a first part ofsecondary content to make decision on when or which second part ofsecondary content to insert in the primary content and perform variousrelated processes described and shown in connection with FIGS. 1-4, 6,9A-D and 12 may also be used.

FIG. 9A is an example of an activation graph, in accordance with someembodiments of the disclosure, and FIG. 9B is an example of a valencegraph, in accordance with some embodiments of the disclosure.

In an embodiment, the emotional state of a viewer is represented using aRussell's circumplex model of emotions, examples of which are displayedin FIGS. 10 and 11 . Other models, in addition to Russell's circumplexmodel may also be used. The models are used to represent user emotionswithin a two-dimensional circle that includes two numbers: one numberfor activation and one number for valence. An example of an activationgraph is displayed in FIG. 9A, and a valence graph is displayed in FIG.9B.

Activation relates to intensity indicating how energized the user iswhen consuming a video, such as the primary or secondary contentdescribed herein. As such, activation is associated with stimuli thatevoke a certain part of the brain with positive high-arousal andnegative low-arousal emotions. It also relates to the strength of theuser's person's inclination to take some action based on the evokedemotion rather than taking no action.

Valence relates to a positive or negative aspect of an emotion. Itindicates the level or degree of how positive or negative the feelingsof the user are, such as in response to consuming a video.

Valence and activation work together to determine the overall emotionresponse of the user. Collectively they represent an emotional impact ona user and determine the influence on a user the emotional impact hascaused for the user to be motivated to take an action based on theiremotional response.

As depicted in FIG. 9C, an average emotional arc is calculated based onthe average valence and activation values. The average emotional arc ofa video is represented by the changing average activation and valence ofthe previous viewers of that video captured over the course of thevideo. The average emotional arc is then used as an estimate for acurrent user. For example, when a current user has not already consumedthe current video, the current user's emotional arc for that video isestimated based on the average emotional arc of the video, with offsetsfor how the user's actual activation and valence during the consumptiondiffer from the average activation and valence of previous viewers atthe same point in the video. The position of the video at which thesecondary content may be best inserted for this current user isdetermined based on the similarity of the viewer's estimated activationand valence to the average before activation and valence values forthose previous viewers who have engaged with that secondary content withthe proviso that the current user's estimated activation and valenceafter the secondary content not become too high or too low from theaverage determined from previous users.

FIG. 9D is an example of a table that represents values of the emotionalarc, in accordance with some embodiments of the disclosure. Theactivation, valence, and emotional arc calculated based on theactivation and valence numbers may be presented in a graphical format,such as in FIGS. 9A-9C, or in a table format such as in FIG. 9D.

In one embodiment, as depicted in FIG. 9D, the emotional arc value is asummation of the balance and the activation value at a particular playposition in the video. As depicted, the balance value at play positionone is “10” while the activation value at the same play position is “5.”As such the calculated emotional arc value at play position one is “15.”In another embodiment, as depicted at play position 20, the emotionalarc value may be a summation of a valence value, activation value, andany values associated with any additional factors, such as a “3” in thisexample, that may be considered to get a complete emotional picture ofthe user's emotions as the user was consuming the video at play position20. For example, the additional factors may include, the current userhaving previously consumed the content, such as the primary content orthe Ad. Since repetition of a previously played content may not have thesame impact at it did when it was played the first time, an offset forthe repetition may be made. Larger offsets may also be made based on thenumber of times the content has been previously consumed, i.e., thelarger the number of times the same content was previously consumed, thelarger the offset. Yet another additional factor may be based on theuser's profile which may be indicative of the user's likes and dislikes.Another additional factor may be based on the consumption history ifthey have liked or disliked similar content during their previousconsumptions. Yet another additional factor may be the time of the day,the day of the week or month, if the user is consuming the contentduring dinner, with family, which browsing other content simultaneouslyon another device etc.

FIGS. 10 and 11 are examples of a list of emotions and their varans frommild to intense and pleasant to unpleasant, in accordance with someembodiments of the disclosure. These emotions include tense, nervous,stressed, upset, side, depressed, sluggish, bored, com, relaxed, serene,contented, happy, elated, enthusiastic, excited, alert, distressed,annoyed, alarm, amused, gloomy, bored, relaxed, satisfied, pleased,frustrated, miserable, and sad to name a few. The type of emotionsdescribed are not limited and other emotions not listed are alsocontemplated.

The models, such as the Russell's circumplex model, may list some of theabove-described emotions on a scale of unpleasant to pleasant, activateddeactivated, intense to mile, and pleasant to unpleasant. Other scalesmay also be used to capture the emotion and its intensity.

FIG. 12 is graph of positive and negative movements in a valence andactivation graph, in accordance with some embodiments of the disclosure.In one embodiment, solid green colored arrows are used for users whohave demonstrated the desired impact of the secondary content. Forexample, if the creator of secondary content has designed the secondarycontent to produce a desired impact, which is defined by the averageemotional arc, then the user's current impact based on their emotionalresponse may be measured against the desired impact and the green arrowswill reflect if such an emotional response was actually attained. Thedashed red colored arrows are used for other users that have notdemonstrated the desired impact.

In one embodiment, the process of generating the green and red arrowsincludes the control circuitry calculating statistics for the users uponwhom the desired impact is observed. In particular, the controlcircuitry captures the average starting activation and valence and theaverage change to the user's activation and valence. The solid greenarrows indicate that the secondary content is most effective for userswho exhibit valence and activation in a middle region, and who uponwatching the secondary content become somewhat “calmer”—as indicated bytheir activation coming down while their valence becomes more positive.In other embodiments, other effects and different levels of valence andactivation, such as a higher valence and activation causing the viewerto be become more energetic or any of the emotions displayed in FIGS.10-11 may be experienced.

It will be apparent to those of ordinary skill in the art that methodsinvolved in the above-mentioned embodiments may be embodied in acomputer program product that includes a computer-usable and/or-readable medium. For example, such a computer-usable medium may consistof a read-only memory device, such as a CD-ROM disk or conventional ROMdevice, or a random-access memory, such as a hard drive device or acomputer diskette, having a computer-readable program code storedthereon. It should also be understood that methods, techniques, andprocesses involved in the present disclosure may be executed usingprocessing circuitry.

The processes discussed above are intended to be illustrative and notlimiting. More generally, the above disclosure is meant to beillustrative and not limiting. Only the claims that follow are meant toset bounds as to what the present invention includes. Furthermore, itshould be noted that the features and limitations described in any oneembodiment may be applied to any other embodiment herein, and flowchartsor examples relating to one embodiment may be combined with any otherembodiment in a suitable manner, done in different orders, or done inparallel. In addition, the systems and methods described herein may beperformed in real-time. It should also be noted that the systems and/ormethods described above may be applied to, or used in accordance with,other systems and/or methods.

What is claimed:
 1. A method comprising: determining a primary emotionalarc of a primary content, wherein the primary emotional arc is based onemotional response data from a plurality of users that have previouslyconsumed the primary content; estimating an emotional response from acurrent user based on the primary emotional arc prior to the currentuser consuming the primary content; determining that a current emotionalresponse from the current user is within a threshold of the estimatedemotional response; and in response to the determination that thecurrent emotional response from the current user is within the thresholdof the estimated emotional response, causing insertion, into the primarycontent during its display on a media device, of secondary content thatis designated for a value of emotional response that is within thethreshold of the estimated emotional response.
 2. The method of claim 1,wherein determining that the current emotional response from the currentuser is within the threshold of the estimated emotional response furthercomprises: determining a value of the primary emotional arc at a firstplay position in the primary content; determining a value of the currentemotional response from the current user at the first play position inthe primary content; and comparing the value of the primary emotionalarc at the first play position with the value of the emotional responsefrom the current user at the first play position to determine whetherthe current emotional response from the current user is within thethreshold of the estimated emotional response.
 3. The method of claim 2,further comprising, causing insertion of the secondary content either atthe first play position or within a predefined time of the first playposition.
 4. The method of claim 1, wherein determining the primaryemotional arc of the primary content comprises: displaying the primarycontent to the plurality of users; receiving emotional response datafrom the plurality of users for the displayed primary content; andgenerating the primary emotional arc based on an average of the receivedemotional response data from the plurality of users.
 5. The method ofclaim 4, further comprising using a camera associated with a mediadevice used by each user, of the plurality of users, to obtain emotionalresponse data, wherein the emotional response data using the cameraassociated with the media device used by each user is obtained frommonitoring by the camera of a gaze of the user during a display of theprimary content.
 6. The method of claim 4, further comprising usingdevices capable of collecting biometric data that are communicativelyconnected with a media device used by each user, of the plurality ofusers, to obtain emotional response data, wherein the emotional responsedata collected by using the devices capable of collecting biometric datathat are communicatively connected with the media device used by eachuser is any one of: user's heartbeat or user's movements.
 7. The methodof claim 1, wherein, determining that the current emotional responsefrom the current user is within the threshold of the estimated emotionalresponse further comprises: monitoring the user's current emotionalresponse during the display of the primary content; generating a currentemotional arc based on the user's current emotional response datacollected from the current user during the display of the primarycontent; and comparing the current emotional arc to the estimatedemotional response to determine if the current emotional arc is within athreshold of the estimated emotional response.
 8. The method of claim 1,further comprising, determining that the current emotional response fromthe current user is not within the threshold of the estimated emotionalresponse; and in response to the determination that the currentemotional response from the current user is not within the threshold ofthe estimated emotional response; determining a current emotional valueof the user at a current play position; and inserting a secondarycontent that is associated with the current emotional value into theprimary content at the current play position.
 9. The method of claim 1,further comprising, associating a value of the primary emotional arc ofthe primary content with one or more emotions.
 10. The method of claim1, further comprising: determining that the current emotional responsefrom the current user is not within the threshold of the estimatedemotional response; and in response to the determination that thecurrent emotional response from the current user is not within thethreshold of the estimated emotional response, providing an offset tothe primary emotional arc of the primary content, wherein the offset isbased on a number of times the current user has consumed the primarycontent.
 11. The method of claim 1, wherein determining a primaryemotional arc of the primary content further comprises: displaying theprimary content to a first group of users that are aligned with a firsttype of interest and to a second group of users that are aligned with asecond type of interest; and determining a first primary emotional arcfor the first group of users and a second primary emotional arc for thesecond group of users based on their respective emotional response data.12. The method of claim 11, wherein, the first type of interest differsor is opposite to the second type of interest.
 13. The method of claim11, further comprising: determining an interest of the current user;comparing the interest with the first type of interest and the secondtype of interest; and in response to determining that the current user'sinterest is aligned with the first type of interest, using the firstprimary emotional arc as the primary emotional arc for the primarycontent.
 14. The method of claim 1, further comprising, using theprimary emotional arc of the primary content as an estimating tool forcomparing the estimated emotional response of the current user prior toconsuming the primary content to an actual response of the current userafter consuming the primary content.
 15. A system comprising:communications circuitry for transmitting a video stream to a mediadevice; and control circuitry to: determine a primary emotional arc ofprimary content, wherein the primary emotional arc is based on emotionalresponse data from a plurality of users that have previously consumedthe primary content; estimate an emotional response from a current userbased on the primary emotional arc prior to the current user consumingthe primary content; determine that a current emotional response fromthe current user is within a threshold of the estimated emotionalresponse; in response to the determination that the current emotionalresponse from the current user is within the threshold of the estimatedemotional response, causing insertion, into the primary content duringits display on the media device, of secondary content that is designatedfor a value of emotional response that is within the threshold of theestimated emotional response; and using the communications circuitry,transmitting in the video stream, the primary content with the insertedsecondary content to the media device.
 16. The system of claim 15,wherein determining that the current emotional response from the currentuser is within the threshold of the estimated emotional response furthercomprises, the control circuitry configured to: determine a value of theprimary emotional arc at a first play position in the primary content;determine a value of the current emotional response from the currentuser at the first play position in the primary content; compare thevalue of the primary emotional arc at the first play position with thevalue of the emotional response from the current user at the first playposition to determine whether the current emotional response from thecurrent user is within the threshold of the estimated emotionalresponse.
 17. The system of claim 16, further comprising, the controlcircuitry configured to cause insertion of the secondary content eitherat the first play position or within a predefined time of the first playposition.
 18. The system of claim 15, wherein determining the primaryemotional arc of the primary content comprises, the control circuitryconfigured to: display the primary content to the plurality of users;receive emotional response data from the plurality of users for thedisplayed primary content; and generate the primary emotional arc basedon an average of the received emotional response data from the pluralityof users.
 19. The system of claim 15, wherein, determining that thecurrent emotional response from the current user is within the thresholdof the estimated emotional response further comprises, the controlcircuitry configured to: monitor the user's current emotional responseduring the display of the primary content; generate a current emotionalarc based on the user's current emotional response data collected fromthe current user during the display of the primary content; and comparethe current emotional arc to the estimated emotional response todetermine if the current emotional arc is within a threshold of theestimated emotional response.
 20. The system of claim 15, furthercomprising, the control circuitry configured to: determine that thecurrent emotional response from the current user is not within thethreshold of the estimated emotional response; and in response to thedetermination that the current emotional response from the current useris not within the threshold of the estimated emotional response, providean offset to the primary emotional arc of the primary content.
 21. Thesystem of claim 15, wherein determining a primary emotional arc of theprimary content further comprises, the control circuitry configured to:display the primary content to a first group of users that are alignedwith a first type of interest and to a second group of users that arealigned with a second type of interest; and determine a first primaryemotional arc for the first group of users and a second primaryemotional arc for the second group of users based on their respectiveemotional response data.
 22. The system of claim 21, further comprising,the control circuitry configured to: determine an interest of thecurrent user; compare the interest with the first type of interest andthe second type of interest; and in response to determining that thecurrent user's interest is aligned with the first type of interest, usethe first primary emotional arc as the primary emotional arc for theprimary content.